Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing

We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.Comment: Published at 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018

On the use of Process Mining and Machine Learning to support decision making in systems design

Research on process mining and machine learning techniques has recently received a significant amount of attention by product development and management communities. Indeed, these techniques allow both an automatic process and activity discovery and thus are high added value services that help reusing knowledge to support decision-making. This paper proposes a double layer framework aiming to identify the most significant process patterns to be executed depending on the design context. Simultaneously, it proposes the most significant parameters for each activity of the considered process pattern. The framework is applied on a specific design example and is partially implemented.FUI GONTRAN

Discovering Concept Maps from Textual Sources

ABSTRACT Concept maps and knowledge maps, often used as learning materials, enable users to recognize important concepts and the relationships between them. For example, concept maps can be used to provide adaptive learning guidance for learners such as path systems for curriculum sequencing to improve the effectiveness of learning process. Generation of concept maps typically involve domain experts, which makes it costly. In this paper, we propose a framework for discovering concepts and their relationships (such as prerequisites and relatedness) by analyzing content from textual sources such as a textbook. We present a prototype implementation of the framework and show that meaningful relationships can be uncovered

Context Aware Trace Clustering: Towards Improving Process Mining Results

Abstract Process Mining refers to the extraction of process models from event logs. Real-life processes tend to be less structured and more flexible. Traditional process mining algorithms have problems dealing with such unstructured processes and generate spaghetti-like process models that are hard to comprehend. An approach to overcome this is to cluster process instances (a process instance is manifested as a trace and an event log corresponds to a multi-set of traces) such that each of the resulting clusters correspond to a coherent set of process instances that can be adequately represented by a process model. In this paper, we propose a context aware approach to trace clustering based on generic edit distance. It is well known that the generic edit distance framework is highly sensitive to the costs of edit operations. We define an automated approach to derive the costs of edit operations. The method proposed in this paper outperforms contemporary approaches to trace clustering in process mining. We evaluate the goodness of the formed clusters using established fitness and comprehensibility metrics defined in the context of process mining. The proposed approach is able to generate clusters such that the process models mined from the clustered traces show a high degree of fitness and comprehensibility when compared to contemporary approaches

Efficient Discovery of Understandable Declarative Process Models from Event Logs

Process mining techniques often reveal that real-life processes are more variable than anticipated. Although declarative process models are more suitable for less structured processes, most discovery techniques generate conventional procedural models. In this paper, we focus on discovering Declare models based on event logs. A Declare model is composed of temporal constraints. Despite the suitability of declarative process models for less structured processes, their discovery is far from trivial. Even for smaller processes there are many potential constraints. Moreover, there may be many constraints that are trivially true and that do not characterize the process well. Naively checking all possible constraints is computationally intractable and may lead to models with an excessive number of constraints. Therefore, we have developed an Apriori algorithm to reduce the search space. Moreover, we use new metrics to prune the model. As a result, we can quickly generate understandable Declare models for real-life event logs

Effect of a checklist on advanced trauma life support workflow deviations during trauma resuscitations without pre-arrival notification

Background Trauma resuscitations without pre-arrival notification are often initially chaotic, which can potentially compromise patient care. We hypothesized that trauma resuscitations without pre-arrival notification are performed with more variable adherence to ATLS protocol and that implementation of a checklist would improve performance. Study Design We analyzed event logs of trauma resuscitations from two 4-month periods before (n = 222) and after (n = 215) checklist implementation. Using process mining techniques, individual resuscitations were compared with an ideal workflow model of 6 ATLS primary survey tasks performed by the bedside evaluator and given model fitness scores (range 0 to 1). Mean fitness scores and frequency of conformance (fitness = 1) were compared (using Student\u27s t-test or chi-square test, as appropriate) for activations with and without notification both before and after checklist implementation. Multivariable linear regression, controlling for patient and resuscitation characteristics, was also performed to assess the association between pre-arrival notification and model fitness before and after checklist implementation. Results Fifty-five (12.6%) resuscitations lacked pre-arrival notification (23 pre-implementation and 32 post-implementation; p = 0.15). Before checklist implementation, resuscitations without notification had lower fitness (0.80 vs 0.90; p \u3c 0.001) and conformance (26.1% vs 50.8%; p = 0.03) than those with notification. After checklist implementation, the fitness (0.80 vs 0.91; p = 0.007) and conformance (26.1% vs 59.4%; p = 0.01) improved for resuscitations without notification, but still remained lower than activations with notification. In multivariable analysis, activations without notification had lower fitness both before (b = -0.11, p \u3c 0.001) and after checklist implementation (b = -0.04, p = 0.02). Conclusions Trauma resuscitations without pre-arrival notification are associated with a decreased adherence to key components of the ATLS primary survey protocol. The addition of a checklist improves protocol adherence and reduces the effect of notification on task performance. © 2014 by the American College of Surgeons Published by Elsevier Inc

Effect of a checklist on advanced trauma life support workflow deviations during trauma resuscitations without pre-arrival notification

Background Trauma resuscitations without pre-arrival notification are often initially chaotic, which can potentially compromise patient care. We hypothesized that trauma resuscitations without pre-arrival notification are performed with more variable adherence to ATLS protocol and that implementation of a checklist would improve performance. Study Design We analyzed event logs of trauma resuscitations from two 4-month periods before (n = 222) and after (n = 215) checklist implementation. Using process mining techniques, individual resuscitations were compared with an ideal workflow model of 6 ATLS primary survey tasks performed by the bedside evaluator and given model fitness scores (range 0 to 1). Mean fitness scores and frequency of conformance (fitness = 1) were compared (using Student\u27s t-test or chi-square test, as appropriate) for activations with and without notification both before and after checklist implementation. Multivariable linear regression, controlling for patient and resuscitation characteristics, was also performed to assess the association between pre-arrival notification and model fitness before and after checklist implementation. Results Fifty-five (12.6%) resuscitations lacked pre-arrival notification (23 pre-implementation and 32 post-implementation; p = 0.15). Before checklist implementation, resuscitations without notification had lower fitness (0.80 vs 0.90; p \u3c 0.001) and conformance (26.1% vs 50.8%; p = 0.03) than those with notification. After checklist implementation, the fitness (0.80 vs 0.91; p = 0.007) and conformance (26.1% vs 59.4%; p = 0.01) improved for resuscitations without notification, but still remained lower than activations with notification. In multivariable analysis, activations without notification had lower fitness both before (b = -0.11, p \u3c 0.001) and after checklist implementation (b = -0.04, p = 0.02). Conclusions Trauma resuscitations without pre-arrival notification are associated with a decreased adherence to key components of the ATLS primary survey protocol. The addition of a checklist improves protocol adherence and reduces the effect of notification on task performance. © 2014 by the American College of Surgeons Published by Elsevier Inc

Effect of a checklist on advanced trauma life support workflow deviations during trauma resuscitations without pre-arrival notification

Background Trauma resuscitations without pre-arrival notification are often initially chaotic, which can potentially compromise patient care. We hypothesized that trauma resuscitations without pre-arrival notification are performed with more variable adherence to ATLS protocol and that implementation of a checklist would improve performance. Study Design We analyzed event logs of trauma resuscitations from two 4-month periods before (n = 222) and after (n = 215) checklist implementation. Using process mining techniques, individual resuscitations were compared with an ideal workflow model of 6 ATLS primary survey tasks performed by the bedside evaluator and given model fitness scores (range 0 to 1). Mean fitness scores and frequency of conformance (fitness = 1) were compared (using Student\u27s t-test or chi-square test, as appropriate) for activations with and without notification both before and after checklist implementation. Multivariable linear regression, controlling for patient and resuscitation characteristics, was also performed to assess the association between pre-arrival notification and model fitness before and after checklist implementation. Results Fifty-five (12.6%) resuscitations lacked pre-arrival notification (23 pre-implementation and 32 post-implementation; p = 0.15). Before checklist implementation, resuscitations without notification had lower fitness (0.80 vs 0.90; p \u3c 0.001) and conformance (26.1% vs 50.8%; p = 0.03) than those with notification. After checklist implementation, the fitness (0.80 vs 0.91; p = 0.007) and conformance (26.1% vs 59.4%; p = 0.01) improved for resuscitations without notification, but still remained lower than activations with notification. In multivariable analysis, activations without notification had lower fitness both before (b = -0.11, p \u3c 0.001) and after checklist implementation (b = -0.04, p = 0.02). Conclusions Trauma resuscitations without pre-arrival notification are associated with a decreased adherence to key components of the ATLS primary survey protocol. The addition of a checklist improves protocol adherence and reduces the effect of notification on task performance. © 2014 by the American College of Surgeons Published by Elsevier Inc

Hierarchical performance analysis for process mining

Process mining techniques use event data from operational and software processes to discover process models, to check the conformance of predefined process models, and to extend such models with information about bottlenecks, decisions, and resource usage. In recent years, the process mining field made huge advances in terms of scalability. In addition, recent work in process discovery supports advanced process model constructs such as subprocesses, recursive structures, cancellation, and various notions of concurrency. Hence, one has to realize that a simple, small, and flat model will not suffice anymore, especially when applied to analyzing software system processes. However, state of the art performance analysis is still typically performed either over the whole process model or at the level of individual activities. There is a lack of formal support for performance analysis on various submodel abstractions while taking into account the execution semantics. This paper presents 1) a framework for establishing precise relationships between events and submodels, taking into account execution semantics; and 2) a novel formalization of existing and novel performance metrics. Our approach enables advanced performance analysis at various submodel abstractions. An implementation is made available, and we demonstrate the advantages of our approach to various software system processes, showing the applicability and advantage with respect to existing techniques